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DETAILED ACTION 

This Final office action is responsive to communication filed March 3, 2010. 
Claims 1 0, 1 1 , 1 5, 1 7, 1 8, 20-22, 24-28, 30 and 31 are currently amended, and claims 
32-37 have been added. Therefore claims 10, 1 1, 15, 17, 18, 20-22, 24-28 and 30-37 
remain pending in this application. 

Claim Objections 

1 . Claim 27 is objected to because of the following informalities: 

2. There is insufficient antecedent basis for the terms "the display" in line 5 of the 
claim and "the boundaries" in line 6 of the claim. 

3. Applicant has deleted the limitations "automatically determining that the content 
displayed within the on screen region includes the textual data" and "automatically 
determining that the displayed content of the on screen region includes underlying 
data... of the selected on screen region." However, the claim includes limitations that 
occur in response to the deleted determination steps, i.e. including "in response to 
automatically determining that the content displayed within the on screen region 
includes the textual data" and in response to "automatically determining that the 
displayed content of the on screen region includes underlying data... of the selected on 
screen region." 

4. Appropriate correction is required. 
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5. Claim 28 is objected to because of the following informalities: It is unclear as to 
what input the claim limitation "interpreting the input" is referring. 

6. The added limitation "determining if the bounded character or word lies within the 
selected on-screen region" is unclear and contradicts the previous limitation "a bounded 
character or word from textual data determined to be included in the on-screen region." 
It is already determined that the bounded character or word is included in the on-screen 
region, thus the determining step is ambiguous. 

7. Appropriate correction is required. 

8. Claim 36 is objected to because of the following informalities: The limitation "any 
subsequent sets of textual characters reside outside the path drawn by the user" is 
unclear. Any textual characters not in the selected region when the region is selected 
are inherently outside the path drawn. The Examiner suggests clarifying the limitation. 
Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

9. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

10. Claims 10, 20-22, 30, 36 and 37 are rejected under 35 U.S.C. 103(a) {current 
application filing date 1/26/2004) as being unpatentable over Denoue et al. (US 
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2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al. (US 
7,042,594 B1 , filing date 3/7/2000) ('Dawe'), further in view of Woolf et al. (WO 
01/42980 A1) ('Woolf), and further in view of Saund etal. (US 2003/0182630 A1) 
('Saund'). 
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With respect to claim 10, Denoue teaches: 

setting a mode of operation to a content capture mode for interpreting user input 
for the purpose of selecting an on-screen region of a display (paragraphs 56, 75-78 and 
86), and receiving a path drawn by a user (paragraphs 54 and 86), the path defining 
boundaries of the selected region of the display (paragraph 59), wherein pixels 
comprising one or more graphical elements being displayed in the selected on-screen 
region (Figure 6, paragraph 67); 

capturing the pixels displayed within the selected on-screen region (paragraphs 
67 and 86-87), and storing the captured pixels (paragraph 89); and 

obtaining context information for the one or more graphical elements by 
automatically applying text recognition (paragraph 79) to an annotation (i.e. freeform 
inks 532, 534, 536 and freeform notes) (paragraphs 57 and 75) {OCR may be applied to 
captured content and captured content may include freeform inks, as seen in figs. 2-5), 

wherein context information is stored in association with the captured data 
(paragraphs 75-76). 

Denoue does not explicitly teach storing captured data in an image file; storing 
the results of text recognition as context information via the computer system; or 
automatically storing context information in association with the image file. 

Dawe teaches a system and method for saving handwriting as an annotation in a 
scanned document (see abstract), in which he teaches: 
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capturing image pixels and storing captured image pixels in an image file (col. 3 
lines 60-63, col. 7 line 61- col. 8 line 4); 

applying text recognition to annotations and storing the results of text recognition 
as context information via the computer system (col. 5 lines 52-60, col. 7 lines 6-20 and 
61-65); and 

automatically storing context information in association with the image file (col. 7 
lines 57-65). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Denoue by the teaching of Dawe because 
storing captured data in an image file; storing the results of text recognition as context 
information; or automatically storing context information in association with the image 
file would enable handwriting to be saved as an annotation, and thus reduce clutter in a 
document presented to a user, while maintaining the information conveyed by the 
handwriting for presentation to the user if desired (Dawe, abstract). 

Further regarding claim 10, Denoue in view of Dawe does not teach receiving a 
user command to set the mode of operation to an annotation mode for interpreting the 
user input for the purpose of annotating the one or more graphical elements and, in 
response to the user command, switching the mode of operation to the annotation mode 
and receiving an annotation drawn by the user on the display. 

Woolf teaches a system and method for annotating an electronic document 
independently of is content (see abstract), in which he teaches receiving a user 
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command to set the mode of operation to an annotation mode for interpreting the user 
input for the purpose of annotating the one or more graphical elements (Figure 4 and 
page 7 lines 5-12), and in response to the user command, switching the mode of 
operation to the annotation mode and receiving an annotation drawn by the user on 
the display (Figures 4 and 5, page 7 lines 24-26), wherein the received annotation is 
implemented using a plurality of tools via a toolbar, the toolbar appearing after 
selecting a button (Figure 4, page 7 lines 5-14). 

Woolf teaches a user interface on a screen that has a toolbar at the bottom of the 
screen (Figure 4). Further, the annotation menu 402 may be considered a toolbar 
because it includes interactive buttons that may be clicked to activate a tool (i.e. ink 
mode, selection mode, highlight mode or erase mode). The annotation menu appears 
in response to tapping or selecting a button. 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Woolf 
because receiving a user command to set the mode of operation to an annotation 
mode for interpreting the user input for the purpose of annotating the one or more 
graphical elements and, in response to the user command, switching the mode of 
operation to the annotation mode and receiving an annotation drawn by the user on 
the display would enable users the ability to annotate pages of a document without 
changing the contents of the document (Woolf, page 2 lines 14-15). 
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Further regarding claim 10, Denoue in view of Dawe and Woolf fails to explicitly 
teach the toolbar (or menu) appearing after selecting an on screen region. 

Saund teaches a method for gestural interpretation in a system for selecting and 
arranging visible material in document images (see abstract), in which he teaches a 
toolbar appearing after selecting an on-screen region (paragraphs 29-30). 

Saund teaches that a user draws a path that encloses or partially encloses at 
least one object. In response to this gesture, a pop-up menu choose box 320 appears. 
The pop-up menu may be considered a toolbar because the select button is an 
interactive button that may be clicked to activate a selection tool. Further, Saund 
teaches that tools are usually provided by menus (paragraph 5). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Saund to 
enable users the ability to enter, select or modify data without pre-selecting an input 
mode (Saund, abstract). 

With respect to claim 20, Denoue in view Dawe, Woolf and Saund teaches 
wherein the context information is stored in such a manner as to be accessible to the 
user for performing at least one of the following: 

searching for said context information; 

displaying the context information simultaneously with the captured image pixels 
(Denoue, paragraphs 75-76; Dawe, col. 8 lines 1-16), and 

navigating a network to a source of the captured image pixels. 
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With respect to claim 21 , Denoue in view Dawe, Woolf and Saund teaches 
wherein the one or more graphical elements represent a first set of one or more textual 
characters, and the method further comprising: obtaining additional context information 
by extracting the first set of one or more textual characters, extracting a second set of 
textual characters displayed in proximity with the first set, and storing the first and 
second sets of textual characters as the additional context information (Dawe, col. 7 
lines 6-20 and lines 61-65; Denoue, paragraph 72), the additional context information 
being automatically stored in association with the image file (Dawe, col. 7 lines 57-65). 

With respect to claim 22, Denoue in view Dawe, Woolf and Saund teaches 
wherein the selected on-screen region is part of a displayed textual region, and the 
graphical elements comprise a first set of one or more textual characters displayed in 
the textual region, the method further comprising: obtaining additional context 
information based on a second set of one or more textual characters displayed in the 
textual region (Dawe, Fig. 4, col. 7 lines 6-20 and lines 61-65; Denoue, paragraph 72), 
the additional context information being automatically stored in association with the 
image file (Dawe, col. 7 lines 57-65) (The words "job" or "awareness" may be 
considered the second set of textual characters). 
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With respect to claim 30, Denoue in view Dawe, Woolf and Saund teaches 
wherein the annotation is stored as originally drawn as additional context information in 
association with the image file (Dawe, col. 7 lines 9-11, 18-20 and 57-65). 

With respect to claim 36, Denoue in view Dawe, Woolf and Saund teaches 
determining that the second sets of textual characters reside within the path drawn by 
the user and any subsequent sets of textual characters reside outside the path drawn 
by the user (Dawe, col. 7 lines 11-16). Dawe teaches that it is determined that the word 
"job" resides within the path drawn. It is inherent that any subsequent characters (i.e. 
characters that are not in the region when the region is selected) will be outside the path 
drawn. 

With respect to claim 37, Denoue in view Dawe, Woolf and Saund teaches 
wherein the received annotation is implemented using a plurality of tools via a toolbar 
(Woolf, Figure 4, page 7 lines 5-14), the toolbar appearing after a stylus has been lifted 
off the display (Saund, paragraphs 29-30). Saund teaches that the pop-up menu (i.e. 
toolbar) appear after a path has been drawn, meaning the stylus has been lifted. 

1 1 . Claim 1 1 , 25 and 26 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 
01/42980 A1) ('Woolf'), and Saund et al. (US 2003/0182630 A1) ('Saund'), as applied to 
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claims 10, 20-22, 30, 36 and 37 above, and further in view of Oppermann et al. (US 
6,334,157 B1, filing date 3/11/1997) ('Oppermann'). 



With respect to claim 1 1 , Denoue in view of Dawe, Woolf and Saund teaches 
claim 10, including a selected on-screen region and automatically storing context 
information in association with the image file (Dawe, col. 7 lines 57-65). 

Denoue in view of Dawe, Woolf and Saund does not teach determining a window 
associated with the selected on-screen region; retrieving an application interface having 
a uniform resource identifier (URI) property from the determined window or parent 
window of the determined window; or obtaining the URI property as additional context 
information. 

Oppermann teaches programmatically providing direct access to user interface 
elements of an application program (see abstract), in which he teaches: 

selecting user interface elements, such as text (col. 8 lines 43 and 49-51) and 
determining a window associated with the selected elements (col. 26 lines 37-40); 

retrieving an application interface having a uniform resource identifier (URI) 
property from the determined window or parent window of the determined window (col. 
25 lines 59-62, col. 28 lines 33-39); and 

obtaining the URI property as additional context information (col. 1 1 lines 1-9, 55- 
60, col. 12 lines 55-60, col. 13 lines 51-60). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Oppermann 
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because determining a window, retrieving an application interface having a uniform 
resource identifier (URI) property from the determined window or parent window of the 
determined window would enable accessibility aids the ability to access and manipulate 
user interface elements of any application program without having prior knowledge of 
the application program or its interface (Oppermann, column 4 lines 27-30). 

With respect to claim 25, Denoue in view of Dawe, Woolf, Saund and 
Oppermann teaches wherein the selected on-screen region includes at least a portion 
of a displayed web page or document (Dawe, Figure 4, col. 4 lines 38-42), and the 
method further comprises: using an application programming interface (API) to query an 
application for additional context information (Oppermann, column 7 lines 36-38), the 
additional context information being automatically stored in association with the image 
file (Dawe, col. 7 lines 57-65), the queried application causing the one or more graphical 
elements to be displayed (Oppermann, column 7 lines 36-45; Denoue, paragraphs 86- 
87). 

With respect to claim 26, Denoue in view of Dawe, Woolf, Saund and 
Oppermann teaches further comprising obtaining a uniform resource identifier (URI) of 
the web page or document as the context information (Oppermann, col. 11 lines 1 -9, 
55-60, col. 12 lines 55-60, col. 13 lines 51-60), the URI being obtained as a result of the 
query using the API (Oppermann, column 7 lines 36-38). 
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12. Claims 1 5, 1 8, 32 and 33 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Denoue et al. (US 2004/0119762 A1, filing date 12/24/2002) 
('Denoue') in view of Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf 
et al. (WO 01/42980 A1) ('Woolf ), and Saund et al. (US 2003/0182630 A1) ('Saund'), as 
applied to claims 10, 20-22, 30, 36 and 37 above, and further in view of Browne et al. 
(US 2004/0135815 A1 , filing date 12/15/2003) ('Browne'). 

With respect to claim 15, Denoue in view of Dawe, Woolf and Saund teaches 
claim 10. 

Denoue in view of Dawe, Woolf and Saund does not teach creating and storing a 
linking structure as the association between the image file and the context information. 

Browne teaches a method and apparatus for image metadata entry (see 
abstract), in which he teaches creating and storing a linking structure as the association 
between the image file and the context information (Figure 12, paragraph 136). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Browne 
because teach creating and storing a linking structure as the association between the 
image file and the context information would enable an easy and efficient method of 
classifying and storing digital images (Browne, paragraph 14). 
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With respect to claim 18, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches wherein the linking structure includes at least one pointer pointing to the stored 
image file or the stored content information (Browne, paragraph 136). 

With respect to claim 32, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches a link to the one or more graphical elements of the selected on-screen region, 
the link comprising one of an implicit link hidden behind the one or more graphical 
elements and an explicit link shown in conjunction with the one or more graphical 
elements (Browne, paragraph 136). (Browne teaches metadata-image associations, in 
which pointers are stored between images and metadata). 

With respect to claim 33, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches a separate linking table or database entry utilizing one or more pointers to the 
one or more graphical elements (Browne, paragraph 136). (Browne teaches metadata- 
image associations, in which pointers are stored between images and metadata). 

13. Claims 32 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 
A1 ) ('Woolf'), and Saund et al. (US 2003/0182630 A1) ('Saund'), as applied to claims 
10, 20-22, 30, 36 and 37 above, and further in view of Hertzfeld et al. (US 
2002/0076109 A1 , filing date 1/25/1999) ('Hertzfeld'). 
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With respect to claim 32, Denoue in view of Dawe, Woolf and Saund teaches 
claim 10. 

Denoue in view of Dawe, Woolf and Saund does not teach a link to the one or 
more graphical elements of the selected on-screen region, the link comprising one of an 
implicit link hidden behind the one or more graphical elements and an explicit link 
shown in conjunction with the one or more graphical elements. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches a link to the one or more graphical elements of the 
selected on-screen region, the link comprising one of an implicit link hidden behind the 
one or more graphical elements and an explicit link shown in conjunction with the one or 
more graphical elements (elements 608 and 616 in Fig. 6, paragraph 38). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Hertzfeld 
to enable recognition of predefined types of text and predefined actions to be 
performed based on the types of text (Hertzfeld, abstract). 

With respect to claim 35, Denoue in view of Dawe, Woolf, Saund and Hertzfeld 
teaches wherein the one or more graphical elements comprise underlying data, 
underlying objects, and underlying associations with remote content (Hertzfeld, Fig. 6, 
paragraph 38). 
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14. Claim 34 is rejected under 35 U.S.C. 103(a) as being unpatentable over Denoue 
et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al. 
(US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 A1 ) 
('Woolf), and Saund et al. (US 2003/0182630 A1) ('Saund'), as applied to claims 10, 
20-22, 30, 36 and 37 above, and further in view of Chan (US 2006/01 00849 A1 ). 

With respect to claim 34, Denoue in view of Dawe, Woolf and Saund teaches 
claim 10. 

Denoue in view of Dawe, Woolf and Saund does not teach displaying the 
obtained context information to the user as one of displaying a property in a window and 
hovering over the one or more graphical elements. 

Chan teaches a system for providing a user bilingual annotations (see abstract), 
in which he teaches displaying the obtained context information to the user as one of 
displaying a property in a window and hovering over the one or more graphical 
elements (paragraphs 1 1 , 49 and 66) (Chan teaches that translated versions of text 
objects may be displayed when the pointer is adjacent to or overlaid by a user's 
pointer). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Chan to 
enable an annotation tool providing annotations in multiple languages, thereby making 
documents accessible to more users (Chan, abstract, paragraph 14). 
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15. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Denoue 
et al. (US 2004/01 1 9762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al 
(US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 A1 ) 
('Woolf'), Saund et al. (US 2003/0182630 A1) ('Saund'), and Browne et al. (US 
2004/0135815 A1 , filing date 12/15/2003) ('Browne'), as applied to claims 15 and 18 
above, and further in view of Newman (US 2003/0101 156 A1, filing date 11/26/2001). 

With respect to claim 17, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches claim 15. 

Denoue in view of Dawe, Woolf, Saund and Browne does not teach wherein the 
linking structure is incorporated in a file separate from the stored image file and the 
stored content information. 

Newman teaches database systems and methods (see abstract), in which he 
teaches wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information (paragraph 16). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Newman 
because wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information would enable additional information about 
image files, such as the origination device, person who created the file, and data/time 
the file was created, to be transmitted and stored along with the image files (Newman, 
paragraph 16). 
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16. Claims 24 and 27 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), and further in view of 
Hertzfeld et al. (US 2002/0076109 A1, filing date 1/25/1999) ('Hertzfeld') and Chan (US 
2006/01 00849 A1). 

With respect to claim 27, Denoue teaches: 

receiving a path drawn on the display by a user via an input device (paragraphs 
54 and 86), the drawn path defining the boundaries of a selected on-screen region of 
the display, the selected on-screen region comprising a plurality of pixels (Figure 6, 
paragraphs 59 and 67); 

capturing the plurality of pixels of the on-screen region (paragraphs 67 and 86- 
87); and 

storing the captured image pixels (paragraph 89); 

extracting a character or word from textual data as context information via the 
computer system (paragraph 79); and 

storing the extracted context information in association with the captured data 
(paragraphs 75-76). 

Denoue does not teach: storing captured data in an image file; in response to 
automatically determining that the displayed content of the on-screen region includes 
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the textual data, automatically extracting a character or word from the textual data as 
context information via the computer system; or storing the extracted context 
information and additional context information in association with the image file via the 
data storage subsystem, such that the context information is accessible when viewing 
the image file. 

Dawe teaches a system and method for saving handwriting as an annotation in a 
scanned document (see abstract), in which he teaches: 

capturing image pixels and storing captured image pixels in an image file (col. 3 
lines 60-63, col. 7 line 61- col. 8 line 4); 

in response to automatically determining that the displayed content of the on- 
screen region includes the textual data, automatically extracting a character or word 
from the textual data as context information via the computer system (col. 7 lines 6-8 
and 15-20); and 

storing the extracted context information and additional context information in 
association with the image file via the data storage subsystem, such that the context 
information is accessible when viewing the image file (col. 7 lines 57-60, col. 8 lines 1- 
16). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Denoue by the teaching of Dawe to enable 
handwriting to be saved as an annotation, and thus reduce clutter in a document 



Application/Control Number: 10/766,319 Page 20 

Art Unit: 2164 

presented to a user, while maintaining the information conveyed by the handwriting for 
presentation to the user if desired (Dawe, abstract). 

Further regarding claim 27, Denoue in view of Dawe does not teach a displayed 
content of the selected on-screen region includes both textual data and underlying data 
comprising at least one of: an executable object, a file, or a link to remote content; or in 
response to determining that the displayed content of the on-screen region includes 
underlying data, automatically extracting a property of the underlying data as additional 
context information via the computer system, the property comprising at least one of: a 
file name, a file identifier, a uniform resource locator (URL), a uniform resource identifier 
(URI), a folder name, or meta-data. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches: 

a displayed content of the selected on-screen region includes both textual data 
and underlying data comprising at least one of: an executable object, a file, or a link to 
remote content (Hertzfeld, steps 602 and 608 in Figure 6, paragraph 38); and 

in response to determining that the displayed content of the on-screen region 
includes underlying data, automatically extracting a property of the underlying data as 
additional context information via the computer system, the property comprising at 
least one of: a file name, a file identifier, a uniform resource locator (URL), a uniform 
resource identifier (URI), a folder name, or meta-data (step 616 in Figure 6, paragraph 
38). 
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It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Hertzfeld 
because the content of the selected on-screen region includes both textual data and 
underlying data comprising at least one of: an executable object, a file, or a link to 
remote content; and in response to determining that the displayed content of the on- 
screen region includes underlying data, automatically extracting a property of the 
underlying data as additional context information, the property comprising at least one 
of: a file name, a file identifier, a uniform resource locator (URL), a uniform resource 
identifier (URI), a folder name, and meta-data would enable recognition of predefined 
types of text and predefined actions to be performed based on the types of text 
(Hertzfeld, abstract). 

Further regarding claim 27, Denoue in view of Dawe and Hertzfeld does not 
teach extracting complete sentences based upon punctuation. 

Chan teaches a system and method for providing bilingual annotations (see 
abstract), in which he teaches extracting complete sentences based upon punctuation 
(paragraph 77). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Chan to 
enable an annotation tool providing annotations in multiple languages, thereby making 
documents accessible to more users (Chan, abstract, paragraph 14). 
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With respect to claim 24, Denoue in view of Dawe, Hertzfeld and Chan teaches: 
digitizing movements of a stylus across the display in order to receive the 

annotation (Denoue, paragraphs 55, 57 and 75); and 

obtaining additional context information based on the received annotation 

(Denoue, paragraph 79), the additional context information being automatically stored in 

association with the image file (Dawe, col. 7 lines 57-65). 

17. Claims 28 and 31 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Dawe et. al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Denoue et 
al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue'), in view of Woolf et al. (WO 
01/42980 A1 ) ('Woolf), and further in view of Hertzfeld et al. (US 2002/0076109 A1 , 
filing date 1/25/1999) ('Hertzfeld'). 

With respect to claim 28, Dawe teaches: 

receiving a path drawn on the display by the input device (col. 5 lines 28-30; col. 
6 lines 48-50), the drawn path defining the boundaries of a selected on-screen region 
of the display, the selected on-screen region comprising a plurality of pixels (col. 6 
lines 42-51); 

capturing the plurality of pixels of the on-screen region (col. 3 lines 60-63, col. 7 
lines 40-42 and lines 61-65, col. 8 lines 1-3); 

storing the captured pixels as an image file via the data storage subsystem (col. 
3 lines 60-63, col. 7 lines 40-42 and lines 61-65, col. 8 lines 1-3), wherein the content 
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displayed within the on-screen region includes at least one of: textual data, an 
executable object, a file, or a link to remote content (col. 4 lines 46-59, col. 7 lines 38- 
40); 

automatically determining via the computer system that the content displayed 
within the on-screen region includes at least one of textual data and other underlying 
data comprising at least one of an executable object, a file, and a link to remote 
content (col. 6 lines 52-53, col. 7 lines 11-20); 

automatically extracting as additional context information: 

a bounded character or word from the textual data determined to be included in 
the on-screen region (col. 7 lines 6-20 and 57-65, col. 8 lines 19-26), and determining 
if the bounded character or word lies within the selected region (col. 7 lines 11-16); and 

storing the context information and the additional context information in 
association with the image file via the data storage subsystem, such that the context 
information is accessible when viewing the image file (col. 7 lines 57-60, col. 8 lines 1- 
16). 

Dawe does not teach setting a mode of operation to a content capture mode for 
interpreting input from an input device for the purpose of selecting an on-screen region 
of a display. 

Denoue teaches systems and methods for freeform pasting (see abstract), in 
which he teaches setting a mode of operation to a content capture mode for 
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interpreting input from an input device for the purpose of selecting an on-screen region 
of a display (paragraphs 56, 75-78 and 86). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Denoue because 
setting a mode of operation to a content capture mode for interpreting input for the 
purpose of selecting an on-screen region of a display would enable the user or other 
users to ability to reselect content in a simplified manner (Denoue, abstract, paragraph 
89). 

Further regarding claim 28, Dawe in view of Denoue does not teach switching the 
mode of operation to an annotation mode and receiving an annotation drawn on the 
display via the input device; or interpreting the input and annotating the content 
displayed within the on-screen region. 

Woolf teaches a system and method for annotating an electronic document (see 
abstract), in which he teaches: 

switching the mode of operation to the annotation mode and receiving an 
annotation drawn on the display via the input device (Figs. 4 and 5, page 7 lines 5-12 
and lines 24-26 and); and 

interpreting the input and annotating the content displayed within the on-screen 
region (Fig. 5, page 5 lines 21-23, page 7 lines 24-26, page 8 lines 27-30). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Woolf 
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because receiving an annotation drawn on the display would enable users the ability to 
annotate pages of a document without changing the contents of the document (Woolf, 
page 2 lines 14-15). 

Further regarding claim 28, Dawe in view of Denoue and Woolf does not teach 
extracting as additional context information, a property of underlying data determined 
to be included in the on-screen region, the property comprising at least one of: a file 
name, a file identifier, a uniform resource locator (URL), a uniform resource identifier 
(URI), a folder name, or meta-data. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches automatically extracting as additional context 
information, a property of the underlying data determined to be included in the on- 
screen region, the property comprising at least one of: a file name, a file identifier, a 
uniform resource locator (URL), a uniform resource identifier (URI), a folder name, or 
meta-data (step 616 in Figure 6, paragraph 38). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Hertzfeld to 
enable recognition of predefined types of text and predefined actions to be performed 
based on the types of text (Hertzfeld, abstract). 
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With respect to claim 31 , Dawe in view of Denoue, Woolf and Hertzfeld teaches 
wherein the annotation is stored as originally drawn as additional context information in 
association with the image file (Dawe, col. 7 lines 9-11, 18-20 and 57-65). 

Response to Arguments 

18. Applicant's arguments with respect to claims 1 0, 1 1 , 1 5, 1 7, 1 8, 20-22, 24-28, 30 
and 31 have been considered but are moot in view of the new ground(s) of rejection. 

19. Applicant's arguments filed March3, 2010, regarding claim 28, have been fully 
considered but they are not persuasive. 

20. Regarding claim 28, Applicant argues that Dawe fails to teach determining if the 
bounded character or word lies within the selected on screen region. Examiner 
disagrees. As explained in the claim objection above, claim 28 requires the bounded 
character or word to be included in the on-screen region, thus it is already determined 
that such is true. Further, Dawe teaches if the region is not determined to be a notation, 
it is determined whether the region includes any handwriting (col. 7 lines 6-20). This 
determination step is equivalent to determining if a bounded character or word (i.e. the 
text "great job") lies within the region. 

Conclusion 

21 . Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See M PEP 
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§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to ALICIA M. LEWIS whose telephone number is 
(571 )272-5599. The examiner can normally be reached on Monday - Friday, 9 - 6:30, 
alternate Friday off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Charles Rones can be reached on 571-272-4085. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/A. M. U 

Examiner, Art Unit 2164 
June 19, 2010 

/Charles Rones/ 

Supervisory Patent Examiner, Art Unit 2164 



